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mis  report  provides  a  synopsis  of  a  six-year  research  and  development  effort  to 
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ers  (CCs)  as  well  as  more  general  information  acquisition  and  processing  capa- 
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NAVTRAEQUIPCEN  IH-307 
SUMMARY 

This  report  provides  a  synopsis  of  a  six-year  research  and  development 
(R&D)  effort  to  investigate  the  feasibility  and  desirability  of  using  computer 
assistance  to  Improve  leadership  and  management  practices  at  Recruit  Training 
Commands  (RTCs).  Information  has  been  derived  on  the  suitability  of  two  com¬ 
puter  systems  and  various  approaches  for  employing  these  systems  in  pursuit  of 
this  goal.  These  approaches  encompass  instruction  and  performance  evaluation 
for  Company  Commanders  (CCs)  as  well  as  more  general  information  acquisition 
and  processing  capabilities  to  assist  in  the  management  of  RTCs. 

Some  of  these  programs  are  ready  for  operational  use,  while  others  would 
require  considerable  development  and  testing  prior  to  operational  implementa¬ 
tion.  There  is  a  variety  of  unique  and  important  contributions  (e.g.,  behav¬ 
ioral  objectives,  training  strategies,  analytic  procedures)  to  be  derived  from 
each  of  the  programs,  regardless  of  their  state  of  operational  readiness. 
Further,  a  new  slant  on  the  concept  concerning  organizational  development  was 
Introduced,  and  the  foundations  for  its  operational  implementation  have  been 
laid.  This  concept  involves  the  introduction  of  a  "seif-correcting"  feature 
into  the  organization.  It  appears  that  this  feature  is  required  for  general 
organizational  effectiveness  and  especially  for  improvement  in  the  areas  of 
leadership  and  management.  An  effort  is  now  underway  to  make  this  feature  a 
permanent  part  of  RTC. 

Preliminary  evidence  is  available  to  indicate  that,  for  the  most  part, 

RTC  Orlando  does  have  the  capabilities  and  motivation  required  to  properly 
use,  maintain  and  extend  the  various  components  of  Computerized  Evaluation 
and  Training  System  (CETS)  and  is  proceeding  vigorously  to  do  so.  This  report 
presents  some  highly  regarded  projects,  utilizing  CETS,  which  have  been 
planned  and  Initiated  under  RTC  initiative  and  control  since  the  termination 
of  most  Naval  Training  Equipment  Center  (NAVTRAEQUIPCEN)  involvement  in  the 
project  a  couple  of  months  ago.  The  extent  to  which  "self -correcting"  orien¬ 
tations,  such  as  exhibited  at  RTC  Orlando,  persist  or  can  be  instituted  where 
it  is  lacking  Is  a  subject  for  further  investigation.  The  existence  of  proper 
user  attitudes  and  orientations  is  especially  critical  to  this  project  because 
the  system.  In  Its  present  state  of  development,  still  requires  considerable 
levels  of  creativity  and  skill  to  exploit  many  of  its  current  and  potential 
capabilities.  These  demanding  personnel  requirements  should  be  reduced  as 
further  development  and  use  of  the  system  establishes  routine  procedures  of 
operation  and  places  greater  burden  for  its  operation  on  the  computer. 

The  evaluation  and  demonstration  of  training  and  cost-effectiveness  is  a 
perennial  problem  In  this  area  of  research,  and  it  was  a  problem  in  this 
project.  The  Informational  programs  and  the  associated  "self-correcting" 
feature  of  CETS  are  viewed  as  a  promising  approach  to  remedying  many  of  these 
problems.  The  present  efforts  at  program  developnent  and  evaluation,  however, 
took  place  under  more  traditional  conditions.  In  spite  of  the  limitations 
Inherent  In  the  traditional  approach,  generally  favorable  and  encouraging 
results  were  obtained. 

The  need  for  a  different  R&D  paradigm  became  evident  when  some  important 
results  were  found  to  be  Inconsistent  from  one  situation  to  another  and  cer¬ 
tain  Issues  critical  to  the  R&D  could  not  be  addressed  using  traditional  R&D 
procedures.  All  things  considered,  there  appears  to  be  adequate  evidence  to 
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merit  further  development  and  use  of  the  CETS  concept  and  associated  products 
In  the  Navy.  In  this  regard,  a  number  of  recommendations  are  made: 

a.  The  CETS  capability  should  be  Implemented  at  one  RTC  and  then,  if 
warranted,  at  the  three  RTCs.  (This  process  would  require  assurance  that  the 
system  Is  operationally  rea<fy  and  that  the  RTCs  have  the  capability  to  proper¬ 
ly  utilize  It.) 

b.  Once  Implemented,  the  self-evaluating  and  self-correcting  capabili¬ 
ties  of  CETS  should  be  utilized  to  continually  Improve  the  system.  This  can 
be  done,  for  example,  by  correlating  the  various  measures  which  are  Intended 
to  be  predictors  of  CC  success  with  the  CC  success  criterion  measures,  and 
eliminating  those  variables  which  do  not  predict  at  an  acceptable  level. 

Also,  new  variables  can  be  added  to  the  system  at  any  time,  and  tested  for 
their  predictive  validity.  The  self-evaluation  process  should  be  a  continuous 
one,  since  changes  In  the  organization  may  alter  the  utility  of  the  CETS 
variables. 

c.  Continued  efforts  to  determine  the  best  and  most  valid  measures  of 
training  effectiveness  should  be  given  high  priority,  and  these  measures 
should  be  Included  In  the  CETS  system  for  evaluation  and  Improvement  of  all 
aspects  of  organizational  operations. 

d.  Alternate  techniques  for  Implementing  and  exploiting  the  present 
programs  should  be  Investigated  as  part  of  the  operational  RAD  effort. 

e.  Further  laboratory  Investigations  should  be  carried  out  to  develop 
Innovative  programs  for  Improving  the  quality  of  leadership/management  actions. 

f.  The  various  products  and  by-products  of  this  project  (e.g.,  training 
programs,  training  strategies,  developmental  procedures,  evaluation  proced¬ 
ures,  performance  evaluation  approaches,  behavioral  objectives,  etc.)  should 
be  considered  for  possible  contributions  to,and  Integration  with,  other  RAD 
efforts  In  the  area. 

g.  A  coordinated  tri-service  RAD  effort  should  be  considered  for  this 
area  wherein  common  programs  are  developed  for  use  across  all  segments  of  the 
military. 

h.  To  determine  the  feasibility,  desirability,  and  general  nature  of 
this  tri-service  effort,  the  conditions  and  requirements  of  all  potential 
users  of  the  programs  should  be  Identified  and  the  nature  of  all  related 
ongoing  and  planned  RAD  and  operational  programs  should  be  reviewed. 
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PREFACE 

Since  its  Initiation  in  early  1973,  this  project  has  been  funded  at 
various  times  by  the  Defense  Advanced  Research  Projects  Agency,  the  Navy 
Personnel  Research  and  Development  Center,  and  the  Office  of  Naval  Research. 

Original  conceptions  for  the  project  were  established  by  Dr.  Jim  Regan  as 
a  test  of  the  PLATO  IV  computer-based  system.  The  project  has  evolved  since 
that  time  to  Include  many  objectives  In  addition  to  those  originally  defined. 

Many  people,  both  military  and  civilian,  have  contributed  to  this  project 
over  the  years.  In  recent  times,  the  outstanding  efforts  and  attitudes  of 
LCDR  William  Sullivan  and  CTRCS  Paul  Zetterholm  of  RTC  Orlando  have  contrib¬ 
uted  invaluably  to  the  project.  Mr.  Marty  Smith,  aided  by  Mr.  Pat  Smith,  ex¬ 
tended  and  refined  the  complex  computer  programs  that  he  developed  in  earlier 
efforts.  Pat  Smith  additionally  performed  many  of  the  critical  day-to-day 
tasks  of  system  operation  and  evaluation.  Mr.  George  Romot  continued  to 
assure  proper  liaison  between  the  NAVTRAEQUIPC  EN  and  RTC  in  the  sensitive  and 
difficult  areas  of  system  Implementation  and  acceptance. 

A  one-year  field  trial  Is  being  planned  in  which  RTC  will  assume  control 
of  most  of  the  operations  of  the  system.  NAVTRAEQUIPC  EN  support  will  be  mainly 
in  the  area  of  software  maintenance.  This  trial  period  will  provide  critical 
Information  on  the  viability  of  the  present  system  at  an  RTC.  Additionally, 
this  trial  period  will  Indicate  the  desirability  of  performing  further  research 
and  development  to  Improve  the  system's  viability.  Information  on  the  ability 
of  RTC  to  effectively  utilize  the  system  was  not  available  from  earlier  field 
tests  because  of  the  extensive  Involvement  of  NAVTRAEQUIPCEN  personnel.  Also, 
some  of  the  more  desirable  capabilities  of  the  system  were  not  available  for 
operational  employnent  until  recently.  Future  plans  for  the  system  will  be 
based  on  the  outcome  of  this  field  test. 
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INTRODUCTION 

The  management  and  leadership  area  is  a  major  source  of  problems  in 
organizations.  Of  particular  concern  are  the  affective  and  interpersonal  com¬ 
ponents  of  this  area;  because  these  components  are  critical  to  organizational 
success,  and  because  they  are  so  resistant  to  efforts  to  improve  them. 

RTCs  have  an  additional  reason  for  interest  in  this  area.  Not  only  are 
effective  leadership  and  management  practices  critical  to  their  mission,  but  a 
large  part  of  their  mission  is  to  instill  such  practices  in  others. 

In  view  of  the  contribution  of  computers  to  other  instructional  areas,  a 
series  of  studies  was  initiated  in  early  1973  to  investigate  the  feasibility 
of  using  a  computer  to  improve  leadership  and  management  practices  at  RTCs. 
Special  emphasis  was  to  be  placed  on  the  interpersonal  and  affective  areas. 
Prior  investigations  in  this  area  were  practically  nonexistant  (with  two  ex- 
1  2 

ceptions  *  ),  so  it  was  necessary  to  begin  the  project  by  identifying  a  varie¬ 
ty  of  ways  in  which  a  computer  might  be  applied  to  this  area.  Several  of 
these  potential  applications  were  developed  and  evaluated,  providing  a  basis 
for  further  research  and  appl ication  in  the  area. 

This  approach  to  the  project  evolved  from  an  originating  plan  to  evaluate 

the  capabilities  of  the  PLATO  IV  computer-based  education  system3  for  im¬ 
proving  the  attitudes,  and  thereby  the  behaviors  of  CCs  at  RTCs.  The  original 
intention  was  to  focus  attention  on  attitudes  toward  general  concepts  (e.g., 

4  5 

job  satisfaction,  respect  for  recruits).  Prior  research  findings  *  ,  however, 
indicated  that  the  instruction  would  be  better  directed  at  attitudes  toward 
the  specific  behaviors  desired  on  the  job. 

The  original  conception  of  the  project  was  further  modified  to  include 
training  on  the  types  of  behaviors  that  a  CC  should  perform.  The  idea  of  this 
approach  was  that,  for  certain  kinds  of  CC  behaviors,  direct  modification  of 
the  behaviors  through  instruction  and  practice  Is  the  best  way  to  change  both 
attitudes  and  job  performance. 


T"^  Sell  man,  P.,  friend,  M.  B,,  and  Kurland,  L.  Simulation  of  the  Initial 
Psychiatric  Interview.  Behavioral  Science,  1966,  2(5),  389-399. 

2.  Vriend,  J.  A.  A  Fully  Equipped  Computer- Assisted  Group  Counseling  Research 
and  Training  Laboratory.  Educational  Technology,  February  1973,  57-60. 

3.  Meller,  D.  V.  Using  PLATO  IV.  CERL,  University  of  Illinois,  Urbana, 
October  1975. 

4.  Ajzen,  I.  and  Fishbein,  M.  Attitudes  and  Normative  Beliefs  as  Factors  In¬ 
fluencing  Behavioral  Intentions.  Journal  of  Personality  and  Social  Psychology, 
1972,  21,  1-9. 

5.  Fishbein.  M.  and  Ajzen,  I.  Attitudes  and  Opinions,  in  P.  Mussen  and  M. 
Rosenzweigh  (Eds.)  Annual  Review  of  Psychology,  1972,  Palo  Alto  Reviews,  Inc, 
1972,  23,  287-544. 
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In  the  process  of  pursuing  these  extended  objectives,  programs  were  de¬ 
veloped  or  suggested  which  appeared  to  be  of  potential  benefit  to  RTCs.  Thus, 
a  new  phase  of  the  project  was  initiated  in  FY  77  to  develop  a  training  and 
evaluation  package  (in  accordance  with  the  joint  requirements  of  the  RTC, 
Orlando,  and  the  NAVTRAEQU IPCEN)  around  one  of  the  previously  developed 
programs,  for  adoption  by  RTC. 

Detailed  descriptions  and  results  of  the  R&D  performed  prior  to  FY  78  are 

presented  In  other  reports® *^*®*®*^*^ The  current  report  presents  an 
overall  view  of  the  entire  project.  However,  an  emphasis  is  placed  on  the 
activities  and  results  from  the  FY  78  effort  to  complete  the  development,  im¬ 
plementation,  and  evaluation  of  the  newly  configured  CETS  that  was  initiated 
in  FY  77. 


5T  Spencer,  6.  J.  and  Hausser,  D.  L.*,  Blaiwes,  A.  S.  and  Weller,  D.  R.  Use 
of  Computer-Assisted  Instruction  for  Interpersonal  Skill  Training  -  A  Pilot 
Study.  1975.  Technical  Report:  NAVTRAEQU  IPCEN  73-C-0133-1. 

7.  Cohen,  J.  L.  and  Fishbein,  M.  Development  and  Research  Utilizing  the 
PLATO  IV  System  for  Company  Commander  Behavioral  Change  Training.  August  1975. 
Technical  Report:  NAVTRAEQU IPCEN  73-C-0129-1. 

8.  Hausser,  D.  L.,  Blaiwes,  A.  S.,  Weller,  D.  R. ,  and  Spencer,  G.  J.  Appli¬ 
cation  of  Computer-Assisted  Instruction  to  Interpersonal  Skill  Training. 

January  1976.  Technical  Report:  NAVTRAEQU IPC EN  74-C-01 00-1. 

9.  Cohen,  J.  L. ,  and  Fishbein,  M.  A  Field  Test  of  the  PLATO  IV  System  for 
Company  Commander  Behavioral  Change  Training.  July  1976.  Technical  Report: 
NAVTRAEQU IPCEN  74-C-0095-1. 

10.  Lukas,  G.,  Blaiwes,  A.  S.,  and  Weller,  D.  R.  Evaluation  of  Human  Rela¬ 
tions  Training  Programs.  January  1977.  Technical  Report:  NAVTRAEQU IPCEN 
75-C-0076-1 . 

11.  Blaiwes,  A.  S.,  Weller,  D.  R.,  and  Romot,  G.  Development  and  Implementa¬ 
tion  of  a  Computerized  Evaluation  and  Training  System  (CETS)  at  a  Recruit 
Training  Command.  March  1978.  Technical  Report:  NAVTRAEQU IPCEN  IH -300. 

12.  Weller,  D.  R.  and  Blaiwes,  A.  S.  Leadership  Dimensions  of  Navy  Recruit 
Company  Commanders  and  Recruit  Morale  and  Performance.  Psychological  Reports, 
1976,  39,  767-770. 

13.  Blaiwes,  A.  S.  and  Weller,  D.  R.  A  Social  Simulator:  Development  and 
Evaluation.  Educational  Technology.  March  1977,  14-20. 


NAVTRAEOUIPCEN  IH-307 
SECTION  II 

PROJECT  OVERVIEW:  FY  73  -  FY  78 

This  section  describes  the  results  of  efforts  to  meet  the  major  project 
objectives  discussed  In  the  foregoing.  At  a  very  general  level  ,  the  major  ob¬ 
jectives  fall  Into  the  following  categories: 

Objective  A  -  Evaluate  the  PLATO  IV  system. 

Objective  B  -  Develop,  demonstrate,  implement,  and  evaluate  computer- 
based  programs  for  instructing  and  evaluating  CCs  in  the  leadership  and 
management  areas. 

OBJECTIVE  A  -  EVALUATE  THE  PLATO  IV  SYSTEM 

General  experiences  with  the  PLATO  IV  system  have  been  documented  in  a 

variety  of  reports1*.  The  most  salient  disadvantages  of  the  system  for  this 
project's  purposes  were  its  costs  of  operation  and  its  unreliability.  These 
problems  seem  to  be  due  to  the  total  dependence  of  the  system  on  long  distance 
telephone  lines;  and  perhaps  to  the  developmental  state  of  the  system  at  the 
time  it  was  used  for  this  project. 

In  the  interest  of  reducing  these  disadvantages,  the  PLATO  IV  system  was 
replaced  by  a  stand-alone  minicomputer.  This  new  system  (described  in  a 

previous  report15)  performed  all  of  the  essential  tasks  previously  performed 
by  PLATO  IV  as  well  as  some  additional  tasks  (e.g.,  card  reader  input,  video 
tape  control)  at  a  lower  overall  cost. 

Certain  features  of  the  PLATO  IV  system  could  not  be  duplicated  with  the 
new  system.  However,  only  minor  use  of  these  features  was  made,  and  their 
absence  was  not  a  serious  loss.  One  PLATO  IV  feature  vrfilch  was  missed  was  the 
inter-user  program  access  and  communication  capability,  which  allowed  a  ready 
interchange  of  programs  and  ideas  among  users.  Some  desirable  applications 
for  this  feature  were  conceived,  and  one  was  partially  developed  and  tested  (a 
computer-moderated  role  play  between  two  students).  The  decision  to  terminate 
the  use  of  PLATO  IV  precluded  further  pursuit  of  these  ideas. 

Other  PLATO  IV  features  could  not  be  duplicated  on  the  new  system  (e.g., 
touch  panel  input,  selective  screen  erase),  but  satisfactory  methods  for  oper¬ 
ating  without  these  features  were  found,  and  their  absence  caused  no  major 
problems. 

The  absence  of  the  TUTOR  language  of  the  PLATO  IV  system  would  have  been 
a  serious  problem,  especially  since  many  programs  had  to  be  translated  to  the 
new  system.  However,  a  new  computer  language  was  developed  which  translated 
much  of  the  TUTOR  code  to  a  form  compatible  with  the  new  system. 


VT,  See  footnotes  6  -  11  on  page  8. 

15.  See  footnote  11  on  page  8. 
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The  new  system  was  superior  to  PLATO  IV  when  it  came  to  the  input  and 
storage  of  large  amounts  of  data.  The  new  system  included  a  high  speed  opti¬ 
cal  card  reader  which  allowed  rapid  entry  of  questionnaire  responses,  and  a 
ten  megabyte  disk  system  which  allowed  storage  of  all  RTC  data.  All  inputs 
to  the  PLATO  IV  system  had  to  be  made  through  a  keyboard,  and  data  storage 
space  allocated  to  each  user  was  much  smaller. 

Another  feature  of  the  current  system  which  was  not  available  on  PLATO  IV 
was  the  automatic  control  of  a  video-tape  player.  The  new  system,  with  its 
high  speed  printer,  also  made  the  production  of  hard  copies  of  programs  and 
displays  much  easier  (PLATO  IV  used  a  slower,  wet-process  copy  machine). 

OBJECTIVE  B  -  DEVELOP,  DEMONSTRATE,  EVALUATE  AND  IMPLEMENT  COMPUTER-BASED 
PROGRAMS 

This  R&D  contributes  a  variety  of  information  and  products  to  CC  training 
and  evaluation  in  the  area  of  leadership  and  management,  as  well  as  to  the 
leadership  and  management  areas  in  general.  These  contributions  stem  from  the 
identification  of  behavioral  objectives  and  training  objectives,  and  extend  to 
the  operational -ready  programs  which  are  directed  at  these  objectives.  These 
programs  are  described  in  detail  in  the  earlier  project  reports.  These  devel¬ 
opments  provide  a  framework  for  training  and  evaluation  in  these  areas  and 
offer  useful  and  innovative  procedures  and  materials  for  implementing  the 
programs. 

DEVELOP  COMPUTER-BASED  PROGRAMS.  Seven  major  areas  of  development  were  identi¬ 
fied  in  the  various  project  reports.  Four  of  these  programs  are  designed  to 
teach  interpersonal  and  managerial  skills  by  taking  various  job-oriented  (simu¬ 
lation)  approaches  to  the  CCs  job.  Two  other  programs  employ  a  Socratic 
approach  to  persuade  the  CC  to  act  in  certain  ways.  A  final  program  provides 
evaluative  and  diagnostic  information  on  the  performance  of  CCs  as  well  as 
serving  as  a  general  management  tool  for  RTCs. 

a.  Case  Study  Program .  ®  One  job-oriented  program  takes  the  student 
through  a  "case  study"  in  which  the  student  selects  a  recruit  chief  petty 
officer  and  then  assigns  tasks  to  the  company.  The  actions  of  the  student  are 
accomplished  by  selecting  from  a  menu  of  choices  which  are  used  to  gather  in¬ 
formation  and  take  actions.  The  performance  of  the  company  improves  or  de¬ 
grades  depending  on  the  quality  of  the  decision  making  activities  of  the  stu¬ 
dents,  just  as  on  the  job. 

b.  Problem  Situation  Program.^  The  job-oriented  characteristics  of  a 
second  program  consist  of  "problem  situations"  wherein  interpersonal  situations 
likely  to  be  encountered  on  the  job  are  described  along  with  alternative  re¬ 
sponses  commonly  made  by  CCs  to  these  situations.  The  student  tries  to  choose 
the  response  that  best  represents  the  concepts  being  taught.  Feedback  is  pro¬ 
vided,  tailored  to  the  particular  response  selected,  concerning  why  it  is 


See  footnote  10  on  page  8. 

17.  See  footnotes  6,  8,  10,  11  on  page  8. 
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correct  or  incorrect.  Tutorial  and  drill  and  practice  modes  are  employed  to 
teach  the  concepts  of  interpersonal  performance  (e.g.,  giving  feedback,  setting 
goals)  that  apply  in  these  situations.  The  job-oriented  aspect  of  the  instruc¬ 
tion  is  derived  from  the  fact  that,  in  contrast  with  more  theoretical  approach¬ 
es,  the  program  teaches  job  skills  by  relating  the  skills  to  specific  appli¬ 
cations  in  specific  job  situations. 

1 8 

c.  Video  Problem  Situations.  Two  further  approaches  to  the  job- 
oriented  mode  utilize  videotape  to  depict  job  situations  better  and  to  permit 
more  natural  responses  to  the  situations.  One  of  these  programs  replaces  the 
verbal  descriptions  of  the  problem  situations  (described  in  the  foregoing) 

wi  r>  videotaped  presentations  of  enacted  and  real-life  interactions  between 
CCs  .irrJ  recruits.  The  computer  controls  the  videotape  display,  showing  appro¬ 
priate  segments  of  tape  at  appropriate  times.  A  drill  and  practice  mode  of 
instruction  is  employed  to  teach  the  lessons  contained  in  the  videotape 
scenarios. 

d.  Video  Feedback  Program. ^ 9  In  the  other  videotape  program,  the  com¬ 
puter  provides  automated  information  displays,  tape  control,  and  data  proc¬ 
essing  as  forms  of  assistance  to  "judges"  in  their  efforts  to  describe  and 
evaluate  student  CC  performance  in  role-played  situations  and  to  feed  back 
this  information  to  the  students. 

20 

e.  Attitude  Discrepancy  Program.  In  the  two  Socratic  approaches,  the 
students  are  provided  information  and  persuasive  arguments  in  efforts  to 
modify  their  behaviors.  One  such  program  determines  discrepancies  between  a 
student's  conceptions  and  intentions  concerning  the  performance  of  specific 
behaviors  (e.g.,  seeking  help  from  other  CCs)  on  the  one  hand,  and  RTC  policy 
regarding  these  behaviors  on  the  other  hand.  The  program  then  addresses  any 
discrepancies  that  were  found  to  exist  by  showing  the  student  that  it  is  in 
his/her  own  best  interest  to  comply  with  the  policy  of  RTC. 

?1 

f.  Evaluation  Information  Program.  The  second  Socratic  approach  re¬ 
veals  any  misconceptions  which  the  student  may  have  concerning  the  process 
used  for  evaluating  CC  performance.  The  student  evaluates  hypothetical  pro¬ 
files  of  CCs,  and  the  computer  uses  regression  analysis  to  determine  the 
weights  which  the  student  placed  on  various  evaluation  criteria.  These 
weights  are  then  compared  with  the  actual  weights  employed  at  RTC  (determined 
by  having  the  actual  evaluators  evaluate  the  same  hypothetical  profiles). 

Any  differences  that  occur  between  these  two  sets  of  weights  denote  miscon¬ 
ceptions  held  by  the  student  regarding  the  evaluation  process.  CCs  are  ex¬ 
pected  to  obtain  better  evaluations  when  their  understanding  of  the  evaluation 
process  becomes  more  accurate. 

22 

g.  Management  and  Evaluation  Program.  This  program  presents  displays 
of  data  which  evaluate  the  performance  of  CCs  and  RTC  in  general.  This 


T5I  See  footnote  1 1  on  page  8. 

19.  Weller,  D.  R.  and  Blalwes,  A.  S.  Computer-Assisted  Judging  and  Feedback 
of  Interpersonal  Skills,  NAVTRAEQU IPC EN,  in  preparation. 

20.  See  footnotes  7  and  9  on  page  8. 

21.  See  footnotes  7  and  9  on  page  8. 

22.  See  footnote  11  on  page  8. 
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program  is  used  by  CCs  to  assess,  diagnose,  and  improve  their  performance;  and 
by  managers  at  RTC  to  identify  areas  that  require  action,  to  determine  the 
particular  action  to  be  taken,  and  to  evaluate  the  effects  of  the  action  after 
it  is  taken. 

The  major  features  of  this  program  are: 

a.  Automatic  storage  and  processing  of  large  amounts  of  data  on  many 
critical  aspects  of  the  RTC  operation  (e.g.,  organizational  measures  of  re¬ 
cruit  performance,  attitudinal  and  evaluation  data  collected  via  surveys). 

b.  Automatic  computation  of  relationships  and  differences  among  all 
categories  of  these  data. 

c.  Automatic  displays  of  these  data  with  isolation  of  the  most  signifi¬ 
cant  relationships  contained  in  the  data. 

d.  Automatic  rankings  of  the  CCs  in  accordance  with  their  performance  on 
any  system  variables  which  are  selected  and  weighted  for  importance  by  RTC 
evaluators. 

DEMONSTRATE  AND  EVALUATE  COMPUTER-BASED  PROGRAMS.  There  are  two  important  as¬ 
pects  to  the  demonstration  and  evaluation  of  the  products  of  this  project. 

They  should  be:  (1)  functionally  effective  such  that  they  appropriately 
accomplish  RTC  functions,  and  (2)  cost-effective  such  that  the  RTC  functions 
accomplished  by  the  new  programs  justify  the  cost  of  their  use. 

Functional  Effectiveness  -  Pre  FY  78.  Of  the  seven  programs  described  in  the 
previous  section,  three  have  been  evaluated  in  previous  reports  (the  problem 
situation  program,  the  attitude  discrepancy  program,  and  the  evaluation  infor¬ 
mation  program).  The  problem  situation  program  was  selected  to  be  the  basis 
of  the  current  program.  A  brief  review  of  the  previous  research  on  this 
program  is  presented  below.  For  results  of  research  on  the  other  two  programs, 
see  the  reports  referenced  in  the  previous  section. 

a.  Skill  Validity:  Previous  studies  have  consistently  demonstrated  the 
validity  of  the  skills  taught  by  the  problem  situation  program.  The  skills 
have  been  correlated  with  several  outcome  measures  such  as  recruit  morale, 
recruit  inspection  scores,  and  recruit  retention  in  the  Navy.  Nearly  all  of 
these  correlations  have  been  positive  and  significant,  indicating  that  the 
skills  are  related  to  important  measures  at  RTC. 

b.  Experimental/Control  Skill  Differences:  Another  consistent  result 
of  the  previous  studies  is  that  the  experimental  subjects  (those  taught  with 
the  program)  are  rated  by  their  recruits  as  performing  the  skills  to  a  greater 
degree  than  control  subjects.  This  positive  training  effect  is  most  pro¬ 
nounced  in  students  who  are  more  motivated  to  do  the  CC  job. 

c.  Experimental /Control  Outcome  Differences:  Results  concerning  experi¬ 
mental/control  differences  on  outcome  measures  (inspection  scores,  morale, 
retention)  have  been  less  consistent.  Experimental  subjects  have  performed 
at  a  higher  level  In  the  majority  of  cases,  but  few  differences  have  been 
significant.  The  obvious  Importance  of  the  outcome  measures  would  make  It 
worthwhile  to  determine  why  positive  results  were  obtained  in  some  instances 
and  not  in  others. 
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Functional  Effectiveness  -  FY  78.  The  program  in  use  during  FY  78  (CETS)  was 
based  on  the  problem  situation  program,  with  the  addition  of  the  video  situa¬ 
tion  program  and  the  management  and  evaluation  program.  The  video  feedback 
program  is  being  evaluated  and  reported  on  in  a  separate  effort. 

First,  results  from  the  problem  situation  and  video  situation  programs 
are  compared  with  the  results  of  previous  studies.  Following  this,  an  evalua¬ 
tion  of  the  management  and  evaluation  program  is  presented. 

Problem  situation  and  video  situation  programs: 

a.  Skill  Validity:  Data  on  all  variables  (e.g.,  skill  performance, 
background,  outcomes,  etc.)  were  collected  on  all  CCs  leading  training  units 
at  RTC,  Orlando  from  October  1977  until  August  1978  (a  detailed  description  of 

23 

all  variables  is  presented  in  a  previous  report  ).  Consistent  with  the  pre¬ 
vious  results,  these  data  showed  that  the  skills  taught  in  the  two  "situation" 
programs  are  valid.  Scores  on  the  ten  skill  areas  were  correlated  with  re¬ 
cruit  inspection  scores,  recruit  attrition  scores,  recruit  morale  scores,  and 
division  officer  evaluation  scores.  These  correlations  are  presented  in 
Table  1.  Of  the  40  correlations  presented,  34  are  in  the  expected  direction. 
Of  the  25  significant  correlations,  23  are  in  the  expected  direction. 

b.  Experimental/Control  Skill  Differences:  All  subjects  for  the  FY  78 
study  were  recent  graduates  of  Company  Commander  School  at  RTC,  Orlando.  Over 
a  period  of  approximately  six  months,  each  class  in  CC  school  was  randomly 
divided  into  control  and  experimental  subjects.  The  experimental  subjects 
received  the  CETS  training  program  in  place  of  some  "shadowing  time"  (i.e., 
watching  active  CCs  do  their  jobs).  This  occurred  between  graduation  and  the 
picking  up  of  their  first  training  unit.  The  control  subjects  received  no 
special  treatment;  Instead,  they  continued  to  engage  in  normal  shadowing 
activities. 

Data  were  collected  for  experimental  and  control  subjects.  The  data  were 
identified  as  coming  from  regular,  experimental,  or  control  CCs  to  facilitate 
statistical  comparisons.  Data  were  collected  on  a  total  of  36  experimental 
and  30  control  subjects. 

The  t-tests  were  performed  on  all  skill  variables  as  assessed  by  the  re¬ 
cruit  survey.  These  results  are  presented  in  Table  2  (numbers  in  Tables  2-7 
are  mean  values).  The  N's  for  these  and  other  tests  exceed  the  number  of  sub¬ 
jects  because  many  subjects  had  lead  more  than  one  training  unit,  and  data 
were  stored  separately  for  each  training  unit. 

None  of  the  t-values  In  Table  2  is  significant,  and  the  differences  are 
about  equally  divided  between  those  favoring  the  controls  and  those  favoring 
the  experimental s.  These  results  are  inconsistent  with  the  previous  research. 

A  possible  reason  for  these  results  concerns  a  policy  change  at  RTC. 
Coincident  with  the  beginning  of  the  FY  78  study,  RTC  decided  to  place  two  CCs 
In  charge  of  each  training  unit  (one  CC  has  lead  each  training  unit  in  the 
past).  Thus,  each  experimental  and  control  subject  was  paired  with  another  CC, 
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usually  one  who  had  lead  training  units  previously.  It  can  be  assumed  that 
the  second  CC  was  a  major  influence  on  the  behavior  of  the  subjects.  This 
influence  could  easily  be  greater  than  the  influence  of  the  experimental  treat¬ 
ment,  and  it  would  tend  to  decrease  or  eliminate  any  experimental  effects 
(especially  where  the  training  is  at  odds  with  traditional  methods  of  leading 
training  units). 

c.  Experimental /Control  Outcome  Differences:  Tables  3  through  7  present 
the  differences  and  t-values  for  the  outcome  variables.  None  of  these  differ¬ 
ences  is  significant.  Recruit  attitude  scores  are  about  evenly  split  between 
those  favoring  controls  and  those  favoring  experimental s.  The  CC  survey  and 
MED  scores  predominantly  favor  the  controls.  The  attrition  scores  all  favor 
the  experimental s,  while  the  division  officer  survey  scores  all  favor  the 
controls. 

These  results  are  disappointing,  and  it  is  reasonable  to  expect  that  the 
addition  of  the  second  CC  in  a  training  unit  would  tend  to  reduct  any  experi¬ 
mental  effect  on  these  variables,  in  the  same  way  as  on  the  skill  variables. 

Management  and  evaluation  program.  This  program  is  considered  separately 
from  the  other  programs  because  it  is  not  strictly  a  training  program.  It  was 
originally  designed  for  the  collection  and  analysis  of  experimental /control 
data,  but  it  soon  became  apparent  that  the  program  had  potential  as  a  general 
management/ eval uation  tool  at  RTC. 

TABLE  1.  SKILL  VALIDITY:  CORRELATIONS  WITH  OUTCOMES 


SKILL 

RECRUIT 

MORALE 

RECRUIT  „ 
ATTRITION* 

RECRUIT 

INSPECTION 

D.0. 

SURVEY 

Concrete 

.58** 

cr> 

o 

. 

i 

.43** 

.16* 

Timely 

.58** 

-.08 

.48** 

.27** 

Clarifying 

.65** 

-.05 

.22** 

.06 

Reasonabl  e 

.04 

.20** 

-.09 

.14* 

Relevant 

.74** 

-.04 

.20** 

.06 

Considerate 

.64** 

-.16* 

-.09 

-.13* 

Human 

.66** 

-.07 

.05 

.00 

Goal  Setting 

.53** 

-.03 

.40** 

.17* 

Instruction 

.67** 

-.11 

.34** 

.16* 

Feedback 

.58** 

-.12* 

.21** 

-.02 

- 325 - 253 - 315 - 252 


a.  Negative  correlations  are  desirable;  i.e.,  high  skill  performance  =  low 
attrition. 

*  Significant  at  .05  level. 

**  Significant  at  .01  level. 
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TABLE  2.  CC  SKILL  SCORES  FROM  RECRUIT  SURVEY 


EXPERIMENTAL 


[(££] 


CONTROL 
N=54 


t-VALUE 


Concrete 

4.14 

4.21 

-.90 

Timely 

4.14 

4.16 

-.36 

Clarifying 

3.76 

3.77 

-.22 

Reasonable 

3.07 

3.02 

1.21 

Relevant 

3.77 

3.79 

-.41 

Considerate 

3.55 

3.54 

.05 

Human 

3.98 

3.96 

.27 

Goal  Setting 

4.05 

4.09 

-.88 

Instruction 

3.89 

3.96 

-.86 

Feedback 

3.62 

3.63 

-.03 

Counsel  ing 

3.81 

3.72 

1.10 

Motivating 

3.66 

3.65 

.13 

Reward/Punish 

3.57 

3.55 

.23 

Total  Score 

3.77 

3.77 

.00 

TABLE  3.  RECRUIT  ATTITUDE  SCORES 

EXPERIMENTAL 

CONTROL 

ATTITUDE 

(N=59) 

t-VALUE 

Respects  CC 

4.38 

4.40 

-.18 

Morale 

3.89 

3.91 

-.23 

Boot  Camp 

2.85 

2.86 

-.21 

Esprit  de  Corps 

4.47 

4.43 

.93 

Reenl istment 

3.39 

3.36 

.58 

Navy 

4.32 

4.31 

.43 

Training 

4.15 

4.11 

.88 

Attitude  Total 

3.92 

3.91 

.29 

TABLE  4. 

CC  ATTITUDE  SURVEY 

EXPERIMENTAL 

CONTROL 

ATTITUDE 

(N*41) 

(N=38) 

t-VALUE 

CC  School 
Job  In  General 
Recruits 
Division  Staff 
RTC 

Survey  Total 
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TABLE  5.  DIVISION  OFFICER  SURVEY 


EXPERIMENTAL 

CONTROL 

VARIABLE 

(N=44) 

(N=44) 

t-VALUE 

Bearing 

4.27 

4.39 

-  .83 

Rel iabil ity 

3.95 

4.14 

-1.42 

Adaptabil ity 

3.87 

4.05 

-1.47 

Initiative 

3.92 

4.14 

-1.31 

Cooperativeness 

4.01 

4.23 

-1.64 

Counsel ing 

3.85 

4.12 

-1 .52 

Management 

3.82 

3.94 

-  .81 

Problem  Solving 

3.80 

3.92 

-  .88 

Discipline 

3.85 

3.98 

-  .87 

Quality  of  TU 

3.86 

4.04 

-1 .40 

Evaluation  of  CC 

3.29 

3.76 

-1.89 

Survey  Total 

3.87 

4.06 

-1.45 

TABLE  6.  RECRUIT  INSPECTION  (MED)  SCORES 

EXPERIMENTAL 

CONTROL 

SCORE 

(N*41) 

(N=39) 

t-VALUE 

Personnel 

3.57 

3.62 

-1.30 

Barracks 

3.81 

3.84 

-  .72 

Locker 

3.07 

3.10 

-  .38 

Infantry 

3.54 

3.59 

-1.16 

Street marks 

12.37 

12.38 

-  .01 

RPBs 

88.4 

87.4 

.14 

MED  Total 

3.50 

3.54 

-1.00 

TABLE  7. 

RECRUIT  ATTRITION 

EXPERIMENTAL 

CONTROL 

VARIABLE 

(N=42) 

(N=38) 

t-VALUE 

Regular  Setbacks 

6.40 

6.57 

-.17 

Special  Setbacks 

13.95 

15.00 

-.87 

Re-recycles 

3.82 

4.36 

-.80 

Total  Setbacks 

20.20 

21.39 

-.63 

Basically,  the  program  allows  rapid  input,  storage,  processing,  and  output  of 
a  wide  variety  of  variables  concerning  RTC.  The  processing  options  Include 
grouping  of  data  in  a  variety  of  ways:  t-tests,  analysis  of  variance,  corre¬ 
lations,  multiple  regressions,  and  the  automatic  ranking  of  CCs  according  to 
their  scores  on  a  wide  variety  of  variables  (selected  and  weighted  by  RTC 
personnel).  A  more  detailed  description  of  this  program  is  given  In  a  pre- 

vious  report.  Examples  of  some  of  the  outputs  are  given  in  Appendix  A. 


TV.  See  footnote  11 
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(For  reasons  of  data  security,  the  data  shown  in  this  appendix  are  not  actual, 
current  data;  but  are  from  a  developmental  test  of  the  system.) 

A  program  of  this  type  has  almost  limitless  applications,  and  RTC  has 
already  used  it  In  a  variety  of  ways.  The  following  items  are  a  sample  of 
some  of  the  recent  findings; 

a.  Training  units  with  two  CCs  (as  compared  to  those  with  one  CC)  have 
higher  recruit  inspection  (MED)  scores,  higher  skill  performance  ratings  of 
CCs,  higher  division  officer  ratings  of  CCs,  greater  recruit  respect  for  CCs, 
but  greater  attrition. 

b.  Recruits  In  training  units  with  high  MED  scores  are  more  favorable  to 
the  current  situation  (higher  morale,  respect  for  CC,  liking  of  RTC),  but  less 
favorable  about  the  future  (esprit  de  corps,  attitude  toward  Navy  in  general, 
intention  to  reenlist). 

c.  CCs  who  earn  higher  MEDs  emphasize  the  technical  side  of  the  job  at 
the  expense  of  emotional  considerations. 

d.  Overall  recruit  attitudes  are  positively  related  to  the  overall  per¬ 
formance  of  the  CETS  skills  by  their  CCs. 

e.  Division  officers  rate  CCs  higher  when  they  tend  to  emphasize  the 
technical  side  of  the  job  and  deemphasize  the  emotional  side. 

f.  Total  attrition  is  Increasing  at  RTC  over  time. 

The  programs  have  been  used  for  other  purposes  at  RTC,  Including  the 
selection  of  the  "sailor  of  the  quarter,"  the  counseling  of  CCs,  and  the 
selection  of  personnel  for  certain  jobs.  Consideration  is  now  being  given  to 
the  possibility  of  obtaining  data  from  CETS  that  would  allow  the  prediction 
of  the  quality  of  a  CC's  performance  prior  to  and/or  very  early  in  his/her 
assignment  as  a  CC  (thereby  providing  a  basis  for  selection  and  special  early 
training  for  CCs). 

Cost-Effectiveness.  The  previous  technical  report  from  this  project25  dis¬ 
cussed  the  cost-effectiveness  of  using  the  CETS  program  at  RTCs.  The  major 
conclusions  from  this  discussion  were: 

a.  A  major  technical  and  administrative  effort  would  be  required  to 
assess  the  cost-related  effects  of  the  program. 

b.  A  rational  approach  to  organizational  development  requires  cost- 
effectiveness  data  for  all  aspects  of  the  operation  of  the  RTCs. 

c.  In  the  absence  of  fundamental  effectiveness  measures  (i.e. ,  perform¬ 
ance  of  recruits  after  RTC),  cost  savings  can  be  assumed  If  the  new  programs 
provide  more  efficient  methods  of  performing  required  tasks.  The  extent  to 
which  a  task  is  required  must  be  a  subjective  judgment  at  this  time.  The 
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tasks  performed  by  the  CETS  system  would  appear  to  be  required  at  RTCs. 

Clearly,  the  advantages  of  the  CETS  system  over  other  methods  of  performing 
the  same  tasks  are  great,  and  easily  justify  the  costs  of  the  system. 

d.  Measures  of  recruit  attitude,  morale,  and  motivation  are  difficult  to 
quantify  in  fiscal  terms;  however,  experimental  results  show  that  such  data 
can  be  quite  meaningful.  These  results  indicate  that  these  measures  might  be 
useful  in  reducing  costs. 

e.  Another  approach  to  cost-effectiveness  assessment  can  be  used  where 
new  programs  are  substituted  for  existing  programs.  Unfortunately,  most  of 
the  tasks  performed  by  the  CETS  programs  either  were  not  performed  previously 
at  RTCs,  or  a  substitution  approach  to  their  evaluation  was  not  feasible.  The 
cost  savings  for  the  few  areas  where  a  substitution  approach  was  used,  however, 
indicate  that  these  functions  alone  can  justify  the  maintenance  costs  of  the 
system.  Further,  several  additional  areas  where  the  system  could  easily  sub¬ 
stitute  for  less  efficient  methods  are  apparent.  Use  of  the  system  in  these 
areas  would  make  its  cost  justification  even  more  compelling. 

These  conclusions  still  appear  to  be  valid.  A  comprehensive  cost-effec¬ 
tiveness  assessment  of  the  CETS  system  continues  to  be  elusive;  however,  addi¬ 
tional  information  that  can  be  subjected  to  a  conventional  cost  analysis  has 
become  available. 

The  purchase  price  of  the  system  was  $35,000;  and  $5,000  per  year  is  re¬ 
quired  for  maintenance.  These  costs  are  variable  depending  on  when  the  system 
Is  acquired  and  which  programs  are  desired  for  implementation.  For  example, 
a  computer  system  costing  as  little  as  $4,000  to  purchase  and  $1,000  per  year 
to  maintain  would  be  adequate  for  operation  of  the  basic  problem  situation 
training  program. 

The  only  direct  cost  in  addition  to  equipment  costs  is  the  cost  of  the 
personnel  required  for  operation  of  the  system.  This  cost  is  also  variable, 
depending  on  the  tasks  assigned  to  the  computer.  A  minimum  of  about  12  man¬ 
hours  per  week  is  required  for  data  collection  and  entry.  However,  several 
hours  per  week  were  devoted  by  RTC  to  survey  administration  before  the  advent 
of  the  CETS  system,  so  most  of  these  12  man-hours  would  be  expended  even  with¬ 
out  CETS.  The  number  of  additional  man-hours  required  for  other  CETS  related 
tasks  also  depends  on  the  extent  to  which  each  activity  can  be  integrated  into 
other  ongoing  activities  at  RTC.  For  example,  the  cost  of  CETS  related  coun¬ 
seling  of  CCs  should  not  be  attributed  to  CETS  if  the  counseling  replaces 
existing  counseling  activities  at  RTC. 

Ideally,  an  education/programming  billet  should  be  provided  for  the  oper¬ 
ation  and  Improvement  of  CETS.  This  billet  should  free  time  from  other  billets 
In  that  It  would  accomplish  tasks  (e.g.,  counseling,  training,  data  analysis) 
that  normally  are  performed  by  a  number  of  other  positions;  and  are  performed 
less  efficiently  due  to  the  lack  of  computer  assistance.  It  Is  very  difficult 
to  ferret  out  the  time  savings  that  might  result  from  the  assignment  of  all 
such  tasks  to  a  single  CETS  billet,  but  the  savings  should  be  considerable. 
Further,  as  more  tasks  are  assigned  to  the  computer  in  the  future,  the  work 
load  on  other  RTC  personnel  should  be  further  relieved. 
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The  only  quantifiable  cost  savings  at  this  time  are:  (1)  about  $3,000 
per  year  for  computer  processing  of  recruit  attitude  data,  and  (2)  about  $3,000 
per  year  for  processing  of  CC  school  scores.  No  dollar  estimates  can  be  placed 
on  the  improved  accuracy  of  the  data  provided. 

Proper  use  of  the  evaluation  program  should  result  in  considerable  cost 
savings.  For  example,  it  is  possible  to  determine  which  variables  are  related 
to  recruit  attrition.  Using  the  system,  it  has  been  found  that  CCs  with  five 
to  ten  years  and  over  20  years  in  the  Navy  have  significantly  greater  total 
"drop  out"  rates  (about  25%)  than  CCs  with  0-5  years  in  the  Navy  (about  16%). 
Findings  such  as  these  suggest  a  potential  for  selecting  and  or  training  CCs 
in  ways  that  lead  to  reduced  attrition.  Such  possibilities  would  be  pursued 
in  future  efforts  on  the  project. 

The  CETS  system  should  be  used  to  determine  the  appropriateness  of  such 
actions,  and  to  monitor  the  effects  of  any  actions  that  are  taken.  A  case  of 
an  actual  application  of  this  sort  can  be  provided.  The  effects  of  operation¬ 
al  changes  at  RTC  are  now  determined  in  a  small  fraction  of  the  time  that  pre¬ 
viously  was  required,  through  use  of  the  automatic  data  processing  capabili¬ 
ties  of  the  system.  For  example,  the  effects  of  changing  from  one  CC  to  two 
CCs  per  training  unit  were  determined  In  a  couple  of  hours.  This  same  task 
would  have  required  at  least  two  man  months  to  accomplish  prior  to  the  intro¬ 
duction  of  the  system  at  RTC. 

Similar  savings  can  be  attributed  to  this  system  for  a  host  of  other  ad¬ 
ministrative  type  tasks.  For  example,  if  the  process  now  being  used  to  select 
CCs  for  jobs  and  honors  (e.g. ,  sailor  of  the  quarter)  were  to  be  attempted 
without  the  aid  of  the  system,  an  estimated  20  additional  hours  per  week  would 
be  required  to  even  approximate  the  operations  now  performed.  The  system  also 
produces  information  critical  to  the  counseling  of  CCs,  which  would  require 
untold  hours  to  acquire  by  more  traditional  means.  (This  is  just  a  small 
sample  of  the  many  ways  the  system  is  and  can  be  used  and  estimates  of  the 
money  saved  due  to  these  operations,  although  considerable.  Is  ill-defined  at 
this  time). 

IMPLEMENT  COMPUTER-BASED  PROGRAMS.  The  CETS  program  has  been  In  continuous 
use  at  RTC,  Orlando  for  approximately  one  year.  RTC  personnel  have  made  ex¬ 
tensive  use  of  the  management  and  evaluation  programs,  and  have  expressed  the 
desire  to  see  these  programs  continue.  Due  to  the  disappointing  results  of 
the  recent  evaluation  of  the  training  programs,  further  development  and  testing 
would  be  required  before  the  training  portions  of  the  CETS  program  are  ready 
for  operational  acceptance  at  RTC. 

One  of  the  concepts  of  the  CETS  evaluation  and  management  programs  is  that 
they  provide  built-in  "self-correctional"  features.  Basically,  CETS  is  de¬ 
signed  to  be  used  by  RTC  as  a  tool  for  organizational  development,  and  it 
appears  critical  that  the  targets  of  this  development  should  Include  CETS 
itself.  Thus,  one  appealing  approach  to  the  enhancement  of  training  In  the 
area  is  to  provide  "self-correcting"  capabilities  (such  as  the  present  ones) 
whereby  the  programs  can  be  gradually  configured  and  continually  reconfigured 
by  operational  personnel  to  meet  operational  requirements. 

To  help  facilitate  such  applications  of  CETS  in  general  areas  of  organi¬ 
zational  management  and  development,  a  "performance  report"  was  prepared  based 
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on  the  data  from  CETS.  This  report,  following  several  more  limited  reports  on 
the  subject,  summarized  some  of  the  findings  (see  previous  section)  and  con¬ 
clusions  derived  from  the  CETS  evaluation  and  management  system  and,  on  the 
basis  of  these  findings,  reconmended  actions  for  possible  implementation  at 
RTC.  Some  examples  of  the  actions  recommended  are: 

a.  Promulgate  the  procedures  employed  with  the  CETS  CC  ranking  program 
to  evaluate  and  select  CCs  and  extend  the  use  of  this  ranking  program  to  other 
areas  at  RTC. 

b.  Establish  the  top  ranked  CCs  and  Divisions  (as  determined  by  CETS)  as 
elite  groups  by  recognizing  and  rewarding  them  in  appropriate  ways. 

c.  Instruct  CCs  to  adjust  their  Instructional  strategies  to  suit  the  re¬ 
quirements  of  particular  Instructional  situations  in  accordance  with  findings 
from  CETS  (e.g.,  teaching  infantry  is  facilitated  by  a  "tougher"  approach  - 
i.e. ,  striving  for  higher  goals  and  setting  shorter  time  limits  for  learning  - 
than  in  teaching  other  MED  areas). 

d.  Capitalize  on  the  resources  at  RTC  by  soliciting  lists  of  recommenda¬ 
tions  for  action  from  the  most  creative  and  capable  personnel  at  RTC.  These 
recommendations  would  be  based  on  these  performance  reports  and  other  data 
that  may  be  gleaned  from  CETS  and  other  sources.  Implement  and  evaluate  (via 
CETS)  the  best  recommendations  and  appropriately  reward  the  people  who  provide 
them. 

This  first  general  performance  report  was  Intended  to  serve  as  a  working 
model  which  could  be  Improved  with  use  at  RTC.  One  of  the  major  benefits  of 
the  performance  report  would  appear  to  be  that  it  allows  organizational  errors 
to  be  detected  and  openly  acted  upon.  This  is  accomplished  by  bringing  conmon 
information  on  organizational  functioning  to  the  attention  of  a  large  number 
of  organizational  managers.  It  seems  that  failures  in  organizational  develop¬ 
ment  efforts  ".  .  .  are  often  ignored  or  denied,  sometimes  remain  completely 
unrecognized,  are  rarely  analyzed,  and  are  almost  never  legitimized  as  valuable 
sources  for  learning.  The  result  ...  is  that  errors  in  (organizational 
development)  .  .  .  theory  and  technology  are  perpetuated,  common  problems  are 
labeled  unique,  and  dominant  practices  remain  unquestioned....  We  must  learn 
how  to  reward  the  personal  vision  and  organizational  commitment  that  fully 
recognize  the  problems  In  creating  change,  that  fully  accept  that  errors  are 
Inevitable  in  these  undertakings,  yet  still  choose  to  risk,  to  persist,  and  to 
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learn  from  the  errors  in  .  .  .  (organizational  development)."  Thus,  CETS 
provides  the  basis  for  a  critical  but  often  missing  aspect  of  an  organization, 
viz.,  the  ability  to  be  "self-correcting." 

RTC  needs  to  be  able  to  develop  and  use  the  performance  report  discussed 
In  the  foregoing,  in  order  to  optimally  implement  the  "sel f-correcting"  capa¬ 
bilities  provided  by  CETS.  A  variety  of  procedures  can  be  used  to  develop  the 
performance  report,  depending  on  the  Informational  needs  of  RTC.  The  proced¬ 
ures  employed  for  the  first  prototype  model  of  this  report  were: 

Mlrvls,  P.  H.  and  Berg,  D.  N.  In  ISR  Newsletter.  Institute  for  Social 
Research.  The  University  of  Michigan,  Spring  1978,  p.  6. 
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a.  Correlations  were  obtained  between  each  variable  of  major  interest 
and  all  other  variables  on  the  system. 

b.  The  t-tests  and  analysis  of  variance  tests  were  obtained  between  and 
among  all  variables  of  major  interest. 

c.  The  significant  relationships  found  with  each  test  were  noted  and 
common  findings  among  the  various  tests  were  determined. 

d.  Tentative  conclusions  and  recommended  courses  of  action  were  derived 
based  on  clusters  of  these  related  findings. 

These  procedures  are  quite  preliminary.  It  is  expected  that  improvements 
In  the  procedure  would  best  evolve  through  continued  use  at  RTC.  Efforts  are 
now  underway  by  RTC  to  secure  support  and  funding  to  make  the  evaluation 
portion  of  CETS  a  permanent  operational  system  at  RTC. 

Preliminary  evidence  Is  available  to  indicate  that  RTC,  in  the  main,  does 
have  the  capabilities  to  properly  use,  maintain  and  extend  the  management  and 
evaluation  programs  of  CETS.  In  the  couple  of  months  since  NAVTRAEQU IPCEN 
support  has  virtually  ended,  RTC,  Orlando  has,  under  their  own  initiative  and 
control,  planned  and  initiated  a  number  of  highly  desirable  projects  based  on 
CETS.  Some  of  these  projects  which  have  come  to  the  attention  of  the  present 
authors  are  listed  In  the  following: 

o  Scheduling  for  Master  Training  Schedule 

o  Plot  variables  over  time  by  T.  0.  average 

o  Convert  DO  surveys  to  annual  evaluation  marks 

o  Correct  and  P-factor  (i.e.,  item  validate)  CC  school  academic  tests 

o  Develop  counselling  problems  on  video-tape  for  CC  School  instruction 

o  Use  test-score  program  developed  for  the  fourth  item  (above)  to  store, 
process  and  compute  grades  for  CC  school  students 

o  Sort  division  scores  by  specific  division  officer  (by  name) 

o  Compare  ASVAB  scores  (total  and  component)  with  attrition 

o  Compare  AFQT  and  Navy  screening  test  scores  for  recruits  with  recruit 
attrition 

o  Schedule  MED  Inspections 
o  Compute  and  file  the  MED  inspection  results 
o  Maintain  CC  rotation  list 

o  Put  recruit  test  bank  on  the  computer  and  generate  random  tests  for 
recruits  quarterly  (or  other  period) 
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o  Develop  and  test  questionnaire  items  for  selecting  the  best  CC  candi¬ 
dates  for  CC  duty  prior  to  their  assignment  at  RTC. 

Trial  and  error  is  bound  to  occur  in  the  process  of  developing  and  imple¬ 
menting  improvements  as  suggested  by  these  sorts  of  programs.  However,  due  to 
the  relative  ease  of  detecting  and  correcting  errors  and  generating  new  and 
innovative  trials  with  CETS,  these  processes  should  be  far  less  costly  and 
more  likely  to  succeed  than  if  they  were  performed  by  the  more  traditional, 
manual  methods.  Thus,  CETS  can  provide  the  means  to  greatly  increase  the 
speed  and  expand  the  scope  of  the  trial  and  error  process,  which  seems  to  be 
Integral  to  the  development  and  operation  of  new  procedures  (as  represented 
In  the  foregoing)  no  matter  who  does  it  or  how  it  is  done.  RTC,  Orlando  has 
demonstrated  an  orientation  and  motivation  at  "self-correction"  prior  to  the 
availability  of  CETS.  These  qualities  are  essential  to  the  proper  utilization 
of  CETS  by  an  organization.  Thus,  RTC,  Orlando  currently  provides  a  desirable 
situation  to  test  CETS.  The  extent  to  which  this  condition  will  be  maintained 
at  the  RTC  and  can  be  prdfluced  in  other  organizations  are  topics  for  further 
research. 

As  an  example  of  a  "trial"  currently  being  conducted  by  RTC,  a  question¬ 
naire  has  been  developed  by  RTC  consisting  of  items  which  distinguished  "good" 
and  "poor"  CCs  once  they  are  at  RTC,  as  determined  from  CETS  data.  A  test  is 
now  being  considered  in  which  Information  on  these  same  items  will  be  obtained 
for  candidate  CCs  prior  to  their  assignments  at  RTC.  If  these  items  distin¬ 
guish  between  good  and  poor  CCs  before  they  are  selected  for  CC  duty,  CC 
selection  can  be  decided  on  a  firmer  basis  and  the  consequent  elevation  of  CC 
quality  could  have  a  tremendous  Impact  on  recruit  training. 

OTHER  RESEARCH  PRODUCTS  WITH  POTENTIAL  FUTURE  USES 

This  research  project  has  been  directed  at  the  training  and  evaluation  of 
CCs  at  RTCs.  However,  the  job  of  CC  and  the  environment  of  RTC  have  much  in 
comnon  with  other  jobs  and  other  organizations.  Thus,  many  of  the  findings 
developed  during  the  course  of  this  project  have  relevance  for  organizations 
In  general,  as  well  as  RTCs  In  particular.  Three  areas  to  which  these  findings 
apply  are:  (a)  leader/manager  training  and  evaluation,  (b)  the  evaluation 
and  management  of  organizational  operations,  and  (c)  training  media. 

a.  Leader  and  manager  training  and  evaluation. 

(1)  Interpersonal  skills  and  behaviors  performed. by  leaders  were 
Identified  and  relationships  between  this  performance  and  the  effectiveness  of 
the  leader  were  established.  The  validation  of  these  skills  and  behaviors 
would  recommend  their  Inclusion  In  training  and  performance  evaluation  programs, 
especially  for  situations  similar  to  that  of  the  CC.  This  information  also 
contributes  to  the  general  literature  on  the  characteristics  of  successful  and 
unsuccessful  leaders. 

(2)  Certain  types  of  attitudes  held  by  leaders  were  found  to  be 
highly  predictive  of  their  behavior,  and  other  types  of  attitudes  were  found 
to  be  unrelated  to  their  behavior.  These  validated  attitudes  can  provide  In¬ 
structional  objectives  and  criteria  for  training  and  evaluation  programs,  as 
well  as  contributing  scientific  information  on  relationships  between  attitude 
and  behavior. 
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(3)  Discrepancies  in  the  conceptions  held  by  the  personnel  in  an 
organization  concerning  behaviors  appropriate  to  the  organization  were  identi¬ 
fied,  quantified,  and  related  to  organizational  position.  This  information 
should  suggest  instruction  and  other  actions  designed  to  reduce  the  discrep¬ 
ancies  existing  in  an  organization. 

(4)  A  framework  of  computer-based  procedures  for  effecting  attitude 
and  behavior  change  in  leaders  was  developed.  The  specifics  of  thes  proced¬ 
ures  were  developed  and  tested,  and  organizational  conditions  that  act  to 
facilitate  or  Inhibit  these  changes  were  noted.  The  seven  basic  approaches 
that  make  up  this  framework  can  be  applied  in  total  or  in  various  combinations 
to  achieve  the  training  objectives  and  adninister  the  evaluation  programs 
identified  In  this  project,  as  well  as  for  additional  purposes  that  may  be 
devised.  This  framework  can  also  suggest  alternate  approaches  to  the  training/ 
evaluation  problem  that  might  improve  upon  the  current  versions. 

(5)  With  certain  procedural  modifications,  the  materials  of  both 
the  problem  situation  program  and  the  Socratic  programs  can  be  applied  with 
non-computer  techniques.  In  fact,  RTC  has  manually  applied  the  behavioral  and 
evaluation  criteria  identified  in  the  process  of  developing  the  Socratic 
programs.  Further,  role  playing  exercises  developed  during  the  production  of 
video  tape  lessons  have  been  used  by  RTC  on  a  routine  basis  before  the  asso¬ 
ciated  computer  programs  were  operational . 

(6)  A  job-oriented  approach  to  instruction  (i.e.,  the  instruction  is 
related  to  specifics  of  the  CC's  job)  was  demonstrated  to  be  feasible  for  the 
leadership/management  area.  This  is  in  contrast  to  the  more  theoretical,  ab¬ 
stract  forms  of  instruction  often  used  in  this  area. 

b.  Management  and  evaluation  of  organizational  operations. 

(1)  Procedures  and  materials  for  involving  managers  in  the  use  of 
scientific  research  methods  as  an  integral  part  of  their  jobs  were  identified 
and  developed,  and  their  operational  feasibility  was  established. 

(2)  The  degree  to  which  valid  and  trusted  measures  of  organizational 
success  are  available  was  Identified  and  the  probable  effects  of  deficiencies 
in  these  measures  were  discerned.  The  absence  of  suitable  success  measures  is 
a  basic  problem  for  RTCs  as  well  as  other  organizations,  and  R&D  is  needed  to 
determine  how  to  obtain  and  use  them. 

(3)  The  findings  already  derived  from  the  evaluation  program  can  be 
used  as  a  basis  for  managerial  actions  at  RTC  for  as  long  as  the  data  remain 
current.  Further  data  can  be  processed  manually;  however,  if  substantial  data 
are  required,  this  task  would  soon  exceed  the  capabilities  of  the  organization. 

c.  Training  media. 

(1)  The  feasibility  and  desirability  of  the  coordinated  uses  of 
video  tape  and  a  computer  in  training  was  demonstrated.  Instruction  using 
life-like  video  tape  situations  was  demonstrated,  along  with  methods  for 
aiding  In  the  evaluation  of  a  person's  interpersonal  actions  as  seen  on  video 
tape  and  the  feedback  of  this  evaluation  to  the  performer. 

(2)  A  capability  was  developed  for  translating  the  TUTOR  language  of 
the  PLATO  IV  system  into  a  language  compatible  with  NOVA  minicomputers. 


23 


NAVTRAEQU IPCEN  IH-307 
SECTION  III 

DISCUSSION  AND  CONCLUSIONS 

There  is  a  sizeable  body  of  research  literature  on  the  effectiveness  of 
training  and  evaluation  for  the  interpersonal  and  affective  areas  of  the 
leader/manager  job.  This  research  indicates  that  these  training  and  evalua¬ 
tion  programs  often  make  little  or  no  detectable  difference  to  organizational 
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success,  or  even  to  job  behavior.  Still,  the  use  of  such  programs  is  wide¬ 
spread,  and  seems  to  be  increasing.  That  high  importance  is  placed  on  Navy 
programs  in  this  area  is  evident  in  the  statements  of  people  in  the  highest 
governmental  positions,  as  well  as  in  the  nature  of  ongoing  R&D  programs.  For 
example,  a  recent  message  from  Admiral  Hayward,  the  newly  appointed  Chief  of 
Naval  Operations  (CNO),  gave  special  emphasis  to  the  need  for  improved  manage¬ 
ment  and  leadership.  One  major  CNO  objective  was: 

"Improving  the  quality  of  leadership  at  all  levels  of  the  Navy 
through  a  program  of  formalized  training  in  leadership  and  manage¬ 
ment  skills  at  those  stages  in  officers'  and  petty  officers' 
careers  where  such  training  is  most  needed  and  can  be  especially 
28 

productive." 

Operational  programs  reflecting  these  concerns  are  also  noted.  For  exam¬ 
ple,  a  new  BuPers  program  called  Leadership  and  Management  Education  and 
Training  (LMET)  is  a  ".  .  .  centralized,  systematic  approach  to  leadership  and 
management  development  ..."  which  is  intended  to  provide  instruction  for 

".  .  .the  whole  Navy.  .  .  .  "29 

Thus,  it  appears  that  training  and  evaluation  programs  should  and  will  be 
implemented  in  this  area.  Further,  these  programs  will  be  developed,  evaluated 
and  selected  on  some  basis  -  intuitive,  empirical,  and/or  experimental.  All 
such  bases  were  employed  in  the  present  R&D.  Taken  as  a  whole,  the  effective¬ 
ness  evaluations  of  the  programs  were  generally  favorable,  especially  when 
compared  with  results  from  similar  efforts.  However,  the  results  were  still 
not  strong  enough  to  constitute  a  major  contradiction  to  the  generalization 
that  program  effectiveness  in  the  area  rests  on  shakey  grounds.  The  major 
reason  for  the  lack  of  more  complete  confidence  in  the  training  programs  is 
the  failure  of  the  most  recent  experimental  evaluation  to  support  the  favorable 
findings  of  the  earlier  evaluations  (more  discussion  of  program  shortcomings  is 
contained  in  the  following).  Neverthel ess ,  overall  results  from  the  present 
project  were  sufficiently  encouraging  to  recommend  further  pursuit  of  this  R&D 
and  implementation  of  its  products  in  operational  settings. 


IT.  Fiedler,  F.  E.  the  Trouble  with  Leadership  Training  is  that  It  Does  Not 
Train  Leaders.  Psychology  Today,  February  1973. 

28.  The  Centerline,  Naval  Training  Equipment  Center,  Orlando,  Florida.  Vol. 
XI,  No.  29,  July  20,  1978. 

29.  Leadership  and  Management  Evaluation  and  Training  (LMET).  All  Han^, 

May  1978.  No.  736,  pp  4-7. 
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Rationale  for  these  recommendations  rests  on  the  favorable  nature  of  most 
of  the  findings  of  the  experimental  evaluations  of  the  training  programs. 
Further,  the  face  validity  of  the  skills  of  these  programs  seems  to  be  good 
(i.e.,  the  skills  being  taught  appear  to  represent  the  skills  the  Navy  wishes 
to  teach)  and  the  methods  for  teaching  the  skills  appear  to  be  effective  and 
efficient.  The  favorable  face  validity  of  the  training  technology  employed  by 
CETS  is  contributed  to  by  its  use  of  automatic  and  individualized  computer- 
assisted  features  with  the  addition  of  computer-controlled  video  tape  presenta¬ 
tions,  all  of  which  are  integrated  into  more  general  programs  of  performance 
evaluation  and  data  analysis. 

This  favorable  assessment  of  face  validity  is  further  supported  by  the 
correlational  data  which  demonstrated  the  empirical  validity  of  the  subject 
matter  of  the  programs,  and  by  the  student  and  organizational  acceptance  of 
both  the  subject  matter  and  procedures  of  the  programs. 

Finally,  the  present  programs  provide  sophisticated,  field  tested  data 
collection  and  processing  capabilities  for  use  by  managers  in  their  organiza¬ 
tional  development  efforts,  including  the  continual  improvement  of  the  training 
and  evaluation  systems. 

These  accomplishments  are  considerable.  However,  they  do  not  obviate  the 
shortcomings  of  the  R&D,  and  these  too  must  be  addressed.  Among  the  most 
apparent  and  damaging  of  the  shortcomings  were  the  failures  to  more  adequately: 
(a)  improve  and  evaluate  the  program  materials;  (b)  obtain  organizational 
emphasis  on  the  use  of  the  programs;  (c)  achieve  organization-wide  indoctrina¬ 
tion  onto  the  programs;  (d)  implement  the  training  in  remedial  and  refresher 
modes. 


Such  deficiencies,  however,  seem  to  be  natural  by-products  of  fete  tradi¬ 
tional  R&D  approach,  and  are  not  likely  to  be  remedied  in  the  current  effort 
(or  in  other  efforts  in  the  area)  until  alternate  R&D  paradigms  are  found. 
Perhaps  the  primary  rationale  for  recommending  further  use  of  CETS  at  RTC  and 
further  development  of  the  CETS  concept  in  the  military  is  that,  with  CETS, 
such  alternative  R&D  paradigms  may  have  been  found.  The  present  programs 
apparently  offer  the  only  embodiment  of  certain  alternative  R&D  procedures 
that  seem  to  be  needed  for  improved  results  in  the  area. 

With  traditional  R&D  procedures,  a  general  program  format  is  selected, 
often  on  intuitive  or  face-validity  bases.  Then,  if  research  is  performed, 
it  essentially  stops  at  the  time  the  system  is  implemented  in  the  operational 
environment.  This  approach  would  seem  to  work  best  vhere  the  continued 
validity  of  the  instruction  is  well  assured.  In  contrast,  in  training  for 
leadership  and  management,  program  validity  is  in  doubt  initially,  and  even 
more  so  in  the  future. 

The  need  for  continuous  evaluation,  revision  and  update  of  training 
programs  is  beginning  to  receive  some  attention  in  the  general  training  field. 
This  attention,  however,  is  mainly  in  the  form  of  proposals,  speculations  and 
token  gestures  and  very  little  seems  to  have  been  done  to  actually  implement 
a  comprehensive  program  to  facilitate  these  processes.  Further,  except  for  a 
few  sporadic,  enlightened  R&D  programs,  this  need  still  does  not  receive  the 
emphasis  it  deserves. 
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One  example  of  such  an  enlightened  program  is  the  Instructional  Systems 

Development  program  at  NAVTRAEQU IPCEN. 30  This  project  takes  the  position  that 
instructor  training  courses  will  include  "...  a  detailed  'how  to'  of  course 
evaluation,  revision  and  update."  The  authors  state  that  .  .  this  require¬ 
ment  is  considered  of  extreme  importance  if  a  viable  weapons  system  training 
program  is  to  be  maintained."  This  requirement  stems  from  changes  in  training 
objectives,  etc.,  as  well  as  ".  .  .  the  normal  tendency  of  a  new  program  to 
deteriorate  with  time  and  personnel  turnover.  ..." 

To  this  end,  this  program  has  initiated  an  R&D  program  for  providing  com¬ 
puter  aids  to  the  process  of  developing  and  modifying  course  materials  to  be 
accomplished  both  before  and  after  a  training  program  has  been  implemented. 

The  present  technology,  however,  is  designed  to  go  beyond  application  to 
single  training  systems.  It  is  intended  to  extend  the  self-correcting  concept 
to  aid  in  managerial  efforts  to  maintain  and  improve  entire  organizations, 
which  include  all  the  individual  training  and  operating  systems  that  comprise 
the  organization. 

What  appears  to  be  needed  in  the  interpersonal  skills  area  is  an  opera¬ 
tionally-attuned  R&D  program  that  continues  as  an  integral  and  routine  part  of 
the  operation  of  the  system.  This  seems  especially  to  be  required  in  this 
area  because  the  effects  of  the  important  variables  of  leader /manager  perform¬ 
ance  are  accessible  only  in  the  ultimate  operational  context  and  over  extended 
periods  of  time. 

An  Illustration  of  how  the  traditional  R&D  paradigm  fosters  the  sorts  of 
shortcomings  noted  in  the  foregoing  Is  provided  by  the  inability  in  this 
project  to  determine  the  influence  on  trained  CC  performance  resulting  from  the 
change  from  one  to  two  CCs  per  training  unit.  With  continuous  R&D,  enough 
data  could  be  collected  to  ascertain  the  effects  of  the  change;  and  with 
operationally-oriented  R&D,  only  the  situation  where  both  CCs  had  the  same 
training  would  need  to  be  investigated  (in  the  present  case,  only  one  CC  was 
trained). 

Another  problem  encountered  in  the  present  R&D  also  indicated  the  need  to 
deviate  from  traditional,  short-term,  non-operational  R&D  approaches.  It  is 
nearly  impossible  (if  not  undesirable)  to  isolate  the  control  group  from  some 
aspect  of  the  experimental  treatment.  Experimental  contamination  stemmed  from 
actions  taken  by  the  RTC  organization  to  transmit  information  to  anyone  who 
needed  it,  even  if  that  information  was  part  of  the  experimental  treatment. 
Contamination  probably  also  occurred  in  the  course  of  casual  interactions  among 
experimental  and  control  students.  Such  conmunications  are  desirable  in  that 
they  indicate  positive  attitudes  toward  the  materials,  and  they  assist  in  the 
Instructional  process.  However,  such  actions  defeat  the  purposes  of  the  ex¬ 
perimental/control  approach. 


3IT  Funaro,  J.  E.  and  Bird,  R.  G.  US  Navy  Instructional  Systems  Development 
Operational  Aviation  Programs.  Interim  Progress  Report  for  US  Air  Force, 
Flying  Training  ISD  Workshop,  Washington,  DC.  Human  Factors  Laboratory, 
NAVTRAEQU  IPCEN,  Orlando,  a,  3  February  1976. 
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Further,  proper  implementation  of  a  program  of  this  nature  seems  to  re¬ 
quire  universal  (not  experimental  vs.  control)  administration.  When  all 
organizational  personnel  have  been  exposed  to  the  same  materials,  the  oppor¬ 
tunities  for  mutual  reinforcement  of  the  instructional  concepts  and  behaviors 
is  greatest.  With  the  present  design,  however,  the  control  group  and  other 
members  of  the  organization  who  were  given  a  very  superficial,  if  any,  intro¬ 
duction  to  the  programs  were  not  in  a  position  to  contribute  this  reinforce¬ 
ment.  On  the  contrary,  these  unindoctrinated  personnel  would  be  expected  to 
disparage  new  ways  of  acting  that  conflict  with  their  own  (such  a  reaction  was 
often  observed  in  students  in  their  initial  responses  to  training  on  CETS). 

Not  only  should  R&D  be  continuous  with  the  training,  but  the  training 
should  also  be  continuous  with  each  student.  Permanent  behavioral  changes  in 
this  area  apparently  do  not  occur  suddenly,  but  rather  undergo  cycles  of 
testing,  rejection,  and  re-adoption  by  the  student  before  they  replace  existing 
lifetime  habits  and  become  stable  components  of  a  person's  behavioral  reper¬ 
toire.  Thus,  as  with  short-term  R&D,  it  is  not  likely  that  short-term  training 
in  this  area  would  denonstrate  consistently  large  on-the-job  effects.  It  is 
more  reasonable  to  expect  the  desired  outcomes  to  result  from  a  series  of 
short-term  training  experiences. 

One  prime  function  of  the  CETS  programs  is  to  provide  diagnostic  informa¬ 
tion  on  CC  performance  so  as  to  provide  a  basis  for  continual  remedial  training 
for  the  CC  throughout  his/her  tour  at  RTC.  Unfortunately,  this  function  was 
developed  too  late  to  receive  adequate  evaluation.  Primary  among  the  advan¬ 
tages  expected  from  the  remedial  training  approach  in  this  situation  is  that 
it  would  allow  CETS  training  to  compete  on  an  equal  basis  with  the  other  con¬ 
siderable  demands  on  a  CC's  time.  Whereas  CETS  training  is  viewed  as  a  means 
for  becoming  a  more  effective  CC,  CCs  have  other  tasks  to  accomplish  at  RTC 
which  are  necessary  if  they  are  to  operate  at  even  a  minimal  level.  Thus, 
these  other  tasks  naturally  take  precedence  over  the  "nice  to  know  but  not 
essential"  perceptions  of  the  CETS  training.  With  the  remedial  approach,  how¬ 
ever,  CETS  training  would  be  Instituted  when  a  CC  is  failing  to  perform  ade¬ 
quately  on  some  aspect  of  the  job  which  CETS  is  designed  to  improve.  This 
would  change  the  status  of  the  CETS  training  from  "nice  to  know"  to  "essential" 
and  greatly  improve  the  motivation  to  learn. 

The  introduction  of  remedial  training  is  only  one  way  that  student  motiva¬ 
tion  could  have  been  improved.  As  suggested  in  the  foregoing,  the  failure  of 
the  organization  to  introduce  personnel  to  the  contents  of  the  programs,  and 
to  emphasize  the  use  of  the  programs  (which  are  attributed  in  large  part  to 
restrictions  in  the  traditional  R&D  paradigm)  could  also  undermine  the  motiva¬ 
tion  of  the  students  to  learn  and  use  the  training  materials.  Correction  of 
these  deficiences  (which  would  be  facilitated  by  CETS)  should  make  a  major  im¬ 
provement  in  training  effectiveness.  Evidence  for  the  particular  importance 
of  student  motivation  in  this  area  is  provided  by  the  current  finding  of  an 
Interaction  between  student  motivation  and  training  effectiveness  (discussed 
in  the  foregoing)  as  well  as  general  confirmation  of  this  finding  in  other  re- 
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search  in  the  area. 


TT  hlnrlchs,  J.  Personnel  Training  in  Handbook  of  Industrial  and  Organiza¬ 
tional  Psychology.  Rand  McNalley,  1976,  edited  by  M.  D.  Dunnette,  p.  856. 
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Thus,  under  this  plan,  the  design  of  the  instructional  and  evaluation 
programs  would  be  tentative  and  initially  of  secondary  importance.  The  criti¬ 
cal  part  of  the  initial  development  would  be  to  provide  mechanisms  suitable 
for  use  mainly  by  non-R&D  personnel  for  continual  modification  and  evaluation 
of  the  programs  and  an  instructional  framework  into  which  the  modifications 
could  be  readily  implemented.  CETS  is  intended  to  provide  such  capabilities. 

Only  with  the  continuous  information  that  a  system  such  as  CETS  would  pro¬ 
vide  can  programs  be  evaluated  and  more  optimal  training  and  evaluation  con¬ 
figurations  be  designed.  Further,  the  availability  of  continuous  feedback  is 
important  not  only  for  the  improvement  of  training  and  evaluation  programs  for 
leaders  and  managers  but  it  also  is  critical  to  any  manager's  effectiveness 
across  a  wide  range  of  tasks  and  at  all  levels  of  an  organization. 

The  importance  of  feedback  to  managers  and  the  problems  involved  in 
getting  and  giving  it  have  been  noted  in  the  general  literature  on  management 
and  leadership  processes,  as  addressed  in  the  following: 

Human  learning  depends  on  feedback  on  the  results  of  one's 
behavior.  Given  the  fragmented,  oral  nature  of  managerial 
work,  leaders  often  get  vague  and  miscellaneous  feedback. 

Because  they  are  generalists,  superficially  involved  with 
many  different  projects,  intimately  involved  with  only  a 
few,  it  is  difficult  for  them  to  give  feedback  as  well. 

Both  giving  and  receiving  feedback,  however,  is  critical 
to  their  effectiveness. 

Building  a  network  of  contacts,  extracting  important  in¬ 
formation  and  passing  it  on,  designing  tasks  to  provide 
useful  feedback,  and  simply  exchanging  information  are 
how  managers  spend  the  majority  of  their  time.... 

Setting  up  feedback  systems,  judgment  systems,  and  systems 
for  dealing  with  stability  and  instability  may  be  a  major 
factor  In  leadership  effectiveness.  Such  structures  are 
the  leader's  methods  of  chopping  up  reality  and  capturing 
vague  intuitions  and  bits  of  Information  to  see  which 
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actions  need  to  be  taken. 

The  evaluation  program  of  CETS  is  an  extremely  efficient  mechanism  for 
"building  a  network  of  contacts,"  "extracting  important  Information,"  "ex¬ 
changing  Information,"  and  other  functions  that  managers  must  perform  in  order 
to  obtain  and  provide  the  feedback  required  by  their  jobs.  To  attempt  to  per¬ 
form  these  managerial  functions  by  more  traditional  means  would  be  exorbitant 
If  not  Impossible.  To  fall  to  perform  these  functions  by  any  means  would  be  to 
omit  some  essential  elements  from  the  managerial  process. 


371  Lombardo,  M.  M.  Cooking  at  Leadership:  Some  Neglected  Areas.  October 
1977,  Office  of  Naval  Research  TR  2,  pp  19-20. 
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APPENDIX  A 

SAMPLES  OF  MANAGEMENT  AND  EVALUATION  SYSTEM  OUTPUTS 

(For  reasons  of  data  security,  the  data  shown  in  this  appendix  are  not  actual, 
current  data;  but  are  from  a  developmental  test  of  the  system.) 

TOTAL  N  EQUALED  401 ,  AND  THE  PRODUCT  MOMENT  CORRELATION  WAS  EQUAL  TO  0.549. 

Y  =  MORALE  X  =  CONSIDERATE 


4.597-1 

4.538-1 

4.479-1  XXX  X  X  X  X 

4.419-1  XXX  XXX 

4.360-1  X  XX  XXXX 

4.301-1  XX  XXXX  XXX 

4.241-1  X  X  X  XX  XXXXX  XXXX  X  X< 

4.182-1  X  XXXXXXX  X  XXXX  X  XXXX  X  X 

4.123-1  X  X  XX  X  XXX  XXXXX  X  XX  X 

4.063-1  X  XX  X  X  XXXX  X  XXX  X  X  XXX  X 

4.004-1  X  X  XX  XXXXX  XX  X  XX  XXX  X  X 

3.945-1  X  X  XXX  X  XX  XXXXXX  XXXX  XX  X  X  X 

3.885-1  X  X  XXX  XXXXXXXXXXXXXXXXX  X 

3.826-1  X  XX  X  X  XXXXX  XXXX  XXX  X  XXXX  XXX  X 

3.767-1  X  X  X  XXX  XX  XXXXXXXX  XXX  X  X 

3.707-1  XX  X  XXXXXX  X  XX  XX  X  XX  X  XX 

3.648-1  XXX  X  X  X  X 

3.589-1  XX  X  X  X  X  XX 

3.529-1  X  XX  XX  XX  XXXX  X 

3.470-1  XX  XXXX  XX  XXX  X 

3.411-1  X  X  XX 

3.351-1  o  X  XX 

3.292-1  X  X  XXX  XX 

3.233-1  XX  X 

3.173-1  X 

3.114-1  X  X 

3.055-1  X  X 

2.995-1 

2.936-1  X 

2.877-1  X  X 

2.817-1 

2.758-1 

2.699-1  X 

2.639-1 

2.580-1  X 

1.832  2.400  2.967  3.535  4.103  4.670 

2.116  2.684  3.251  3.819  4.387 

The  increment  for  X  Is  0.047;  for  Y  It  is  0.059. 

Figure  A-l.  Product-Moment  Correlation  Output 
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CHOOSE  AN  OFT  ION  AND  A  VARIABLE  NUMBER 


VARIABLE 

NAME 


ASSIGNED 


1. 

IN  USE 

CONSIDERATE 

(27) 

50% 

2. 

IN  USE 

MORALE 

(36) 

50% 

RANKING 

COMPANY  COMMANDER 

SCORE 

T.  U.'S  LEAD 

#1 

AG1 

A 

4.516500 

2  T.U.'S 

#2 

BTCS 

B 

4.457001 

1  T.U.'S 

#3 

AMS1 

C 

4.381500 

1  T.U.'S 

#4 

PHI 

D 

4.355500 

2  T.U.'S 

#5 

ABCS 

E 

4.354501 

1  T.U.'S 

#6 

AE2 

F 

4.347000 

1  T.U.'S 

#7 

BTC 

G 

4.326251 

2  T.U.'S 

#8 

HTC 

H 

4.316001 

1  T.U.'S 

#9 

SK2 

I 

4.312751 

2  T.U.'S 

10 

BMCS 

J 

4.311001 

1  T.U.'S 

11 

NCI 

K 

4.305000 

1  T.U.'S 

12 

YN1 

L 

4.301500 

1  T.U.'S 

13 

ENCS 

M 

4.298000 

1  T.U.'S 

14 

OSC 

N 

4.291500 

1  T.U.'S 

15 

MMCS 

0 

4.268500 

1  T.U.'S 

16 

TM2 

P 

4.266500 

1  T.U.'S 

17 

BTC 

Q 

4.257251 

2  T.U.'S 

18 

DK1 

R 

4.252501 

1  T.U.'S 

19 

EMI 

S 

4.246000 

1  T.U.’S 

20 

MMCS 

T 

4.241501 

1  T.U.'S 

Figure  A-2.  CC  Ranking  Program  Output 
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CATEGORY  VARIABLE  BEING  TESTED  IS  "MORALE" 

GROUP  VARIABLE  t-TEST  BEING  PERFORMED  ON  IS  "CC  RANK" 


#04 

by 

#05; 

t 

= 

1.196  &  df  = 

58. 

by 

#06; 

t 

= 

0.987  &  df  = 

149. 

by 

#07; 

t 

= 

1.019  &  df  = 

156. 

by 

#06; 

t 

= 

1.430  &  df  = 

28. 

by 

#09; 

t 

s 

1.374  &  df  = 

4. 

#05 

by 

#06; 

t 

s 

-1 .633  &  df  = 

207. 

by 

#07; 

t 

= 

-0.719  4  df  * 

214. 

by 

#08; 

t 

s 

0.397  4  df  * 

86. 

by 

#09; 

t 

s 

0.031  4  df  = 

62. 

#06 

by 

#07; 

t 

= 

1.120  4  df  = 

305. 

by 

#08; 

t 

= 

1.712  4  df  = 

177. 

by 

#09; 

t 

— 

0.592  4  df  = 

153. 

#07 

by 

#08; 

t 

s 

0.960  4  df  = 

184. 

. 

by 

#09; 

t 

s 

0.269  4  df  = 

160. 

#08 

by 

#09; 

t 

c 

-0.171  4  df  = 

32. 

MEAN  FOR 

CC 

RANK 

a 

E4  WAS  4.2530 

MEAN  FOR 

CC 

RANK 

= 

E5  WAS  3.8856 

MEAN  FOR 

CC 

RANK 

a 

E6  WAS  3.9604 

MEAN  FOR 

CC 

RANK 

a 

E7  WAS  3.9207 

MEAN  FOR 

CC 

RANK 

a 

E8  WAS  3.8591 

MEAN  FOR 

CC 

RANK 

= 

E9  WAS  3.8812 

ANOVA  TABLE 

BY  CC  RANK  FOR  MORALE 

SOURCE 

ss 

df 

MEAN  SQUARE 

TOTAL 

37.699 

401. 

BETWEEN 

0.523 

5. 

0.105 

WITHIN 

37.176 

395. 

0.094 

Figure  A-3.  t -Test/Ana lysis  of  Variance  Output 
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YOU  HAVE  CHOSEN  "MORALE"  AS  THE  DEPENDENT  VARIABLE 


1). 

MORALE 

R 

3 

1.000  (N 

s 

325.) 

2). 

RECRUIT  ATTITUDES 

R 

s 

0.824  (N 

= 

325.) 

3). 

MOTIVATING 

R 

s 

0.745  (N 

= 

325.) 

4). 

RESPECTS  CC 

R 

= 

0.677 

[N 

= 

325.) 

5). 

INSTRUCTION 

R 

= 

0.669 

[N 

= 

325. 

6). 

REWARD  &  PUNISHMENT 

R 

= 

0.665 

(N 

= 

325.) 

XX  7). 

COUNSELING  (RC) 

R 

= 

0.657 

N 

= 

325. 

8). 

CLARIFYING 

R 

= 

0.653 

(N 

= 

325.) 

si- 

RELEVANT 

R 

= 

0.631 

N 

= 

325.) 

io . 

BOOT  CAMP 

R 

= 

0.614 

N 

= 

325.) 

XX  11). 

CONCRETE 

R 

= 

0.583 

N 

= 

325. 

12). 

TIMELY 

R 

= 

0.583 

N 

= 

325.) 

XX  13). 

CONSIDERATE 

R 

= 

0.581 

(N 

= 

325.) 

14). 

HUMAN 

R 

s 

0.580 

N 

= 

325.) 

15). 

FEEDBACK 

R 

= 

0.574 

N 

= 

325.) 

16). 

TRAINING 

R 

= 

0.555 

N 

s 

325.) 

17). 

NAVY 

R 

s 

0.538 

[N 

s 

325.) 

18). 

ESPRIT  DE  CORPS 

R 

= 

0.533 

N 

= 

325.) 

19  . 

GOAL  SETTING 

R 

= 

0.496 

N 

= 

325.) 

20  . 

RE-ENLIST 

R 

= 

0.360 

[N 

= 

325.) 

21). 

T.  GROUP 

R 

= 

0.187 

[N 

= 

325.) 

22). 

MED  PERSONNEL 

R 

s 

0.162 

N 

s 

319.) 

TYPE 

IN  THE  NUMBER  NEXT  TO  THE 

INDEPENDENT  VARIABLE 

TO  INCLUDE  IN/DELETE  FROM  THE  REGRESSION  ANALYSIS,  /OR 
TYPE  'V  TO  RETURN  TO  THE  LIST  OF  DEPENDENT  VARIABLES,  /OR 
TYPE  'N'  TO  SEE  THE  NEXT  PAGE  OF  CORRELATIONS,  /OR 
TYPE  'P'  TO  SEE  THE  PREVIOUS  PAGE  OF  CORRELATIONS 


Figure  A-4.  Multiple  Correlation  Output 
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MULTIPLE  REGRESSION  FOR  MORALE 

with 

CONCRETE 
CONSIDERATE 
COUNSELING  (RC) 

REGRESSION  STATISTICS: 


MULTIPLE  R  =  0.69558 
R  SQUARED  =  0.48383 
STANDARD  ERROR  =  0.33478 


ANALYSIS  OF  VARIANCE 

SS 

OF 

MS 

F 

REGRESSION 

33.72176 

3. 

11.24059 

100.29560 

RESIDUAL 

35.97595 

321. 

0.11207 

REGRESSION  EQUATION  IS: 

MORALE  =  -0.67244  (CONSTANT) 

+  CONCRETE  X  0.34632  (BETA  =  0.27406) 

+  CONSIDERATE  X  0.39555  (BETA  =  0.22378) 

+  COUNSELING  (RC)  X  0.41451  (BETA  s  0.29551 ) 

Figure  A-5.  Multiple  Regression  Output 


SELECTED  FROM  COMPANY  COMMANDER  SURVEY.  PLOT  CATEGORY  IS:  "JOB  IN  GENERAL 
BEGINS  WITH  T.U.  #3001  AND  ENDS  WITH  T.U.  #3060  FOR  DIVISION  #7 


onoooonoiOtfxoioiooniovooiomnnnmoMonooN 
cnQOoonouMOtcuHOoniotooionnnmntonocQOOio 
onOooonouxoiouKcorouMootcnnnnntoQoiAOinou) 
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Figure  A-6.  Historical  Plot  of  Variable 
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